On Reinforcement Learning of Control Actions in Noisy and Non-Markovian Domains

If reinforcement learning (RL) techniques are to be used for \real world" dynamic system control, the problems of noise and plant disturbance will have to be addressed. This study investigates the e ects of noise/disturbance on ve di erent RL algorithms: Watkins' Q-Learning (QL); Barto, Sutton and Anderson's Adaptive Heuristic Critic (AHC); Sammut and Law's modern variant of Michie and Chamber's BOXES algorithm; and two new algorithms developed during the course of this study. Both these new algorithms are conceptually related to QL; both algorithms, called P-Trace and Q-Trace respectively, provide for substantially faster learning than straight QL overall, and for dramatically faster learning (by up to a factor of 200) in the special case of learning in a noisy environment for the dynamic system studied here (a pole-and-cart simulation). As well as speeding learning, both the P-Trace and Q-Trace algorithms have been designed to preserve the \convergence with probability 1" formal properties of standard QL, i.e. that they be provably \correct" algorithms for Markovian domains for the same conditions that QL is guaranteed to be correct. We present both arguments and experimental evidence that \trace" methods may prove to be both faster and more powerful in general than TD (Temporal Di erence) methods. The potential performance improvements using trace over pure TD methods may turn out to be particularly important when learning is to occur in noisy or stochastic environments, and in the case where the domain is not well-modelled by Markovian processes. A surprising result to emerge from this study is evidence for hitherto unsuspected chaotic behaviour with respect to learning rates exhibited by the well-studied AHC algorithm. The e ect becomes more pronounced as noise increases.